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CLAIMS 

What is claimed is: 

5 1 . A method of real-time rendering in a gaming environment to create an 

interactive experience, comprising: 

capturing a plurality of real-time video streams of a local participant from a 
plurality of camera viewpoints; 

generating real-time renderings of said local participant based on said plurality of 
10 real-time video streams by applying a new view synthesis technique, said real-time 
renderings taken from a perspective of a remote participant located remotely in said 
gaming environment; and 

sending said real-time renderings to said remote participant for viewing within said 
gaming environment. 

15 

2. The method of Claim 1, further comprising: 

incorporating said real-time renderings comprising real-time photo-realistic 
features of said local participant into a graphically generated avatar, said graphically 
generated avatar representing said local participant in said gaming environment; and 
20 sending said graphically generated avatar incorporating said real-time renderings to 

said local participant for viewing within said gaming environment. 

3. The method as described in Claim 1, wherein said generating real-time 
renderings of said local participant further comprises: 

25 generating image based renderings of said local participant as said real-time 

renderings, wherein said new-view synthesis technique comprises an image-based visual 
hull technique. 
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4. The method as described in Claim 1, wherein said generating real-time 
renderings of said local participant further comprises: 

generating a three-dimensional geometric model of said local participant from said 
plurality of real-time video streams as said real-time renderings. 

5 

5. The method as described in Claim 1. wherein said generating real-time 
renderings of said local participant further comprises: 

rendering said real-time renderings from a second location of said remote 
participant with respect to a first location of said local participant within a coordinate 
10 space of said gaming environment. 

6. The method as described in Claim 5, further comprising: 

enabling a change in at least one of said first and second locations by allowing said 
local participant and said remote participant to navigate through said coordinate space; 
15 and 

modifying said real-time renderings to reflect said change in at least one of said 
first and second locations. 

7. The method as described in Claim 1, further comprising: 

20 blending said real-time renderings within a synthetic rendering of said gaming 

environment; and 

displaying said real-time renderings that is blended within said gaming environment 
at a display viewable by said second participant to enable real-time video communication 
for interactive gaming between said local participant and said remote participant within 
25 said gaming environment. 

8. The method as described in Claim 1, wherein said gaming environment 
comprises a three dimensional gaming environment. 
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9. The method as described in Claim 1, further comprising: 
personifying said graphically generated avatar to said local participant by 

illustrating real-time emotion of said local participant through said video image stream 
comprising representative said real-time photo-realistic features of said local participant. 

10. The method as described in Claim 1, further comprising: 

generating another real-time renderings of said local participant based on said 
plurality of real-time video streams by applying said new view synthesis technique, said 
another real-time renderings taken from a perspective of another participant located 
remotely in said gaming environment; and 

sending said another real-time renderings to said another participant for viewing 
within said gaming environment. 

11. A method of real-time rendering in a gaming environment to create an 
interactive experience, comprising: 

capturing a plurality of real-time video streams of a local participant from a 
plurality of camera viewpoints; 

generating a video image stream in real-time of said local participant based on said 
plurality of real-time video streams by applying a new view synthesis technique, said 
yideo image stream rendered from a perspective of a remote participant located remotely 
in said gaming environment; and 

incorporating said video image stream comprising real-time photo-realistic features 
of said local participant into a graphically generated avatar, said graphically generated 
avatar representing said local participant in said gaming environment. 

12. The method as described in Claim 11, wherein said generating said video 
image stream further comprises: 
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rendering said video image stream from a second location of said remote 
participant with respect to a first location of said local participant within a coordinate 
space of said gaming environment. 

13. The method as described in Claim 12, further comprising: 

enabling a change in at least one of said first and second locations by allowing said 
local participant and said remote participant to navigate through said coordinate space; 
and 

modifying said video image stream to reflect said change in at least one of said first 
and second locations. 

14. The method as described in Claim 11, further comprising: 

blending said graphically generated avatar incorporating said video image stream 
within a synthetic rendering of said gaming environment; and 

displaying said graphically generated avatar incorporating said video image stream 
within said gaming environment at a display viewable by said remote participant to enable 
real-time video communication for interactive gaming between said local participant and 
said remote participant within said gaming environment. 

15. The method as described in Claim 11, further comprising: 
personifying said graphically generated avatar to said local participant by 

illustrating real-time emotion of said local participant through said video image stream 
comprising representative said real-time photo-realistic features of said local participant. 

16. The method as described in Claim 1 1, wherein said new-view synthesis 
technique comprises an image-based visual hull technique. 
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17. The method as described in Claim 11, wherein said generating said video 
image stream further comprises: 

generating a three dimensional geometric model of said local participant based on 
said plurality of real time video streams by applying said new view synthesis technique; 
and 

generating said video image stream of said local participant from said three 
dimensional geometric model. 

1 8. The method as described in Claim 1 1 , further comprising: 

generating a real-time audio output from audio of said local participant to enable 
real-time audio communication in said gaming environment between said local participant 
and said remote participant. 

1 9. The method as described in Claim 1 1 , further comprising: 

varying a total of said plurality of camera viewpoints when capturing said 
plurality of real-time video streams of said local participant to vary image quality in said 
plurality of video image streams. 

20. The method as described in Claim 1 1 , further comprising: 

performing non-photorealistic processing of said plurality of video image streams 
resulting in a non-photorealistic cartoon rendering of said plurality of video image streams. 

21. A gaming station comprising: 

a plurality of video cameras for recording real-time video streams of a local 
participant from multiple sample viewpoints surrounding said local participant; 

a new view synthesis module for generating a plurality of output real-time 
renderings of said local participant, by applying a new view synthesis technique to said 
real-time video streams, wherein said output real-time renderings are taken from a 
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plurality of locations within a coordinate space of a three dimensional gaming 
environment. 

22. The gaming station of Claim 2 1 , further comprising: 

an interface module for incorporating a plurality of input video image streams 
comprising real-time photo-realistic features of a plurality of observing participants into a 
plurality of graphically generated avatars representing said plurality of observing 
participants, and blending said plurality of graphically generated avatars within a 
synthetic rendering of said three dimensional gaming environment. 

23. The gaming station as described in Claim 22, wherein said interface module 
receives a plurality of input audio streams in real-time associated with said plurality of 
observing participants for mixing into a single audio stream that is broadcast through a 
local speaker. 

24. The gaming station as described in Claim 22, further comprising: 

a display for viewing said plurality of graphically generated avatars representing 
said plurality of observing participants to enable real-time view communication between 
said local participant and said plurality of observing participants within said three 
dimensional gaming environment. 

25. The gaming station as described in Claim 2 1 , further comprising a contour 
extraction module for extracting said local participant from a physical background. 

26. The gaming station as described in Claim 21, further comprising a means for 
capturing an audio stream in real-time of said local participant. 

27. A computer system comprising: 
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a processor; and 

a computer readable memory coupled to said processor and containing program 
instructions that, when executed, implement a method of real-time rendering in a gaming 
environment to create an interactive experience, comprising: 

capturing a plurality of real-time video streams of a local participant from a 
plurality of camera viewpoints; 

generating real-time renderings of said local participant based on said plurality of 
real-time video streams by applying a new view synthesis technique, said real-time 
renderings taken from a perspective of a remote participant located remotely in said 
gaming environment; and 

sending said real-time renderings to said remote participant for viewing within said 
gaming environment. 

28. The computer system of Claim 27, wherein said method further comprises: 
incorporating said real-time renderings comprising real-time photo-realistic 

features of said local participant into a graphically generated avatar, said graphically 
generated avatar representing said local participant in said gaming environment; and 

sending said graphically generated avatar incorporating said real-time renderings to 
said local participant for viewing within said gaming environment. 

29. The computer system as described in Claim 27, wherein said generating real- 
time renderings of said local participant in said method further comprises: 

generating image based renderings of said local participant as said real-time 
renderings, wherein said new-view synthesis technique comprises an image-based visual 
hull technique. 

30. The computer system as described in Claim 27, wherein said generating real- 
time renderings of said local participant in said method further comprises: 
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generating a three-dimensional geometric model of said local participant from said 
plurality of real-time video streams as said real-time renderings. 

3 1 . The computer system as described in Claim 27, wherein said generating real- 
time renderings of said local participant in said method further comprises: 

rendering said real-time renderings from a second location of said remote 
participant with respect to a first location of said local participant within a coordinate 
space of said gaming environment. 

32. The computer system as described in Claim 3 1 , wherein said method further 
comprises: 

enabling a change in at least one of said first and second locations by allowing said 
local participant and said remote participant to navigate through said coordinate space; 
and 

modifying said real-time renderings to reflect said change in at least one of said 
first and second locations. 

33. The computer system as described in Claim 27, wherein said method further 
comprises: 

blending said real-time renderings within a synthetic rendering of said gaming 
environment; and 

displaying said real-time renderings that is blended within said gaming environment 
at a display viewable by said second participant to enable real-time video communication 
for interactive gaming between said local participant and said remote participant within 
said gaming environment. 

34. The computer system as described in Claim 27, wherein said gaming 
environment comprises a three dimensional gaming environment. 
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35. The computer system as described in Claim 27, wherein said method further 
comprises: 

personifying said graphically generated avatar to said local participant by 
illustrating real-time emotion of said local participant through said video image stream 
comprising representative said real-time photo-realistic features of said local participant. 

36. The computer system as described in Claim 27, wherein said method further 
comprises: 

generating another real-time renderings of said local participant based on said 
plurality of real-time video streams by applying said new view synthesis technique, said 
another real-time renderings taken from a perspective of another participant located 
remotely in said gaming environment; and 

sending said another real-time renderings to said another participant for viewing 
within said gaming environment. 
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